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NASA/ASEE MANAGEMENT SCIENCE 
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KSC MANAGEMENT TRAINING SYSTEM PROJECT 

FINAL REPORT- SUMMER 1993 

Faculty Fellow: Josd. A. Sepulveda, Ph.D., P.E. 

Associate Professor 

Department of Industrial Engineering and Management Systems 
University of Central Florida 
Orlando, Florida 32816 


1 O OBJECTIVES 

The stated objectives for the Summer of 1993 were: 

1. Review the Individual Development Plan Surveys for 1994 in order to automate the analysis of 
the Needs Assessment effort. 

2. Develop and implement evaluation methodologies to perform ongoing program-wide course-to- 
course assessment. This includes. 

. Propose a methodology to develop and implement objective, performance-based, 

awssmenl instruments for each training effort. . 

. Mechanize course evaluation forms and develop software to facilitate the data gathering, 

analysis and reporting processes. . , 

. Implement the methodology, forms, and software in at least one training course or 

seminar selected among those normally odered in the summer at KSC. 

Section 2 of this report addresses the work done in regard to the Individual Devdopmori -Pla» 
Surveys for 1 994. Section 3 presents the methodology proposed to devdop and ""foment objec , 
performance-based, assessment instruments for each training course offered at KSC. 
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2 0 THE INDIVIDUAL DEVELOPMENT PLAN S URVEYS FOR 1994 


Section 2 of this report deals with the first objective, the Individual Development Plan Surveys 
for 1994 The information stored in the Personnel Training Developmental System (PTDS) was 
tra nsferred to a Windows platform and processed using FoxPro for Windows. Sect, on 2.1 the 

SOI- commands used to query the data bases and produce the requested reports As an example the first 
Lge “port is presented in the next pages. The full reports, already delivered to NASA are 
considered parttf this Final Report. Section 2.2 discusses issues associated to training cost and priority 
and presents recommendations for further research. 


2.1 Commands Used to Query the Data Bases 

2.1.1 Report #1 -- Courses by Type of Training by Organization 


SELECT DISTINCT Reporll.unit, Reportl.tot, Report l.course_num„ 
COUNT(Reportl.course_num), SUM(Reportl.cost), Courses. title,, 
Traintyp.type, Units.unitname; 

FROM Report 1, Courses, Units, Traintyp; 

WHERE Courses.course num = Report l.course_num; 

AND Units.unit = Rcportl.unil; 

AND Traintyp. tot = Reportl.tot; 

AND Reporll.unit = "AC"; 

GROUP BY Rcportl courscjium; 

ORDER BY Report 1. tot; 


SSt C E ^roRT e ^rUX,x FROM Rcportl WIDTH 0 COLUMN NOOVERWRITE 
REPORT FORM report l .frx TO PRINTER NOCONSOLE 


Report # 2 - Type of Training by Organization 


SELECT DISTINCT Rcportl.unil, Reportl.tot, COUNT(Reportl.course_num)„ 
SUM(Report 1 .cost), Traintyp.type, Units.unitname; 

FROM Rcportl, Courses, Units, Traintyp, 

WHERE Courscs.course_num = Report l.course_num; 

AND Units.unit = Report 1 unit; 

AND Traintyp. tot = Reportl.tot, 

GROUP BY Reporll.unit, Reportl.tot; 

ORDER BY Reporll.unit, Reportl.tot; 


CREATE^RETORT rcp<M2.frx FROM Rcport2 WIDTH 5120 COLUMN NOOVERWRITE 
llmlr FORM rcport2Ttx TO PRINTER NOCONSOLE NOEJECT 
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Page 


1 


07/15/93 1994 individual Development Plans 


Organization: Ac - comptroller 

type Of Training: 10 -- Executive and Management 


Omrsc 

K1094 

K2176 

K514 

K545 

K6140 

K6155 

K6339 

K6427 

K6715 

K6777 

K753 

K0164 

K8176 

K971G 

K994 1 

K9942 


(>nr« ’I i!lr 

come se exec run roi. issue 

SCIENCE TECH AMO rtlPt.IC POLICY 
NASA MANAGEMENT EPOCATION PROG 
EXEC PEVE1 .orMEtlT SEMINAR 
RES MGT F.PNC PROGRAM 
MANAGEMENT PEV SEM 
HUMAN ELEMENT (THE) 

COMMUNICATION h INEl.UF.NCE 

HFP UTPATE : MGMT PRACTICES *< ORGAN. CLIMATE 
UI-inF.nPTAtlPftK* DUS IMF SS TOT, ICY & OPS. 
MANAGEMENT PEVELO PI 1EI IT SEMINAR 
ADMINISTRATION OF PUBLIC POLICY 
CROSSING DEPARTMF.I IT LIMES 
TASK MANAGEMENT (NASA) 

SEMTMAR ON MANAGERIAL COMPETENCIES 
EXECUTIVE PROJECT MANAGEMENT 


Subtotals for type io 


Executive and Manag^tnpnt 


Number of Requests 
1 
1 

12 

1 

S 

1 

3 

3 
1 
2 

4 
3 
9 
3 
1 

2 

52 


$0 . 00 
$ 1 , 775 . 00 
$ 24 , 000 . 00 
$ 2 , 175.00 
$ 4 , 465.00 
$ 1 , 350.00 
$ 3 , 750.00 
$ 630.00 
$ 670 . 00 
$ 6 , 600 . 00 
$ 0.00 
$ 5 , 325.00 
$ 4 , 140.00 
$ 0.00 
$ 0.00 
$ 0 . 00 

$ 54 , 880 . 00 
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07/15/93 


Page 


1994 1 ndi vicinal Development Plans 

Organization: ac -- comptroller 


Tj|IC of Training: Number of Request 


10 


Executive and Management: 

57 

20 

-- 

Superv \ sot y 

68 

3,T 

-- 

Ena 1 n^rr inq - General 

3 

3M 

-- 

Fmjlnporlmi - Apronpncp 

2 

30 

-- 

Frutinppr Inq - Elect: r lea .1 /Electronic 

1 

3T 

-- 

Engineering - Systems, Industr ial 

1 

4A 

-- 

Public or Husiness Administration 

46 

4D 

— : 

Finance, Account ing. Auditing 

89 

4C 

— 

Personnel 

9 

4n 

— 

EFO, flare or Ethnic Studies 

2 

4E 

-- 

Procurement 

97 

40 

-- 

Policy, Program, or Management Ana lysis 

7 

4M 


Security or Investigation 

2 

4U 

-- 

Program and Project Management 

25 

4Z 

-- 

Other Admin 1st rat 1 ve 

1 

5A 

-- 

Systems, Safety and Area Access Training 

4 

sn 

-- 

Academic Course Work 

101 

6A 

-- 

naslc Clerical 

0 

60 

-- 

Inter-personal and office Management 

11 

6C 

-- 

Administrative Systems 

10 

6I> 

-- 

Other clerical 

2 

0A 

-- 

Orientation 

1 

HE 

-- 

Nets/puallty Circles 

3 

er 

-- 

Productivity 

1 

oo 

-- 

Professional /Personal Development 

13 

OH 

— 

Cornmunicat ion Skills 

25 

07. 

-- 

Other General 

1 

9 A 

-- 

Telecommunicat Ions, Networking 

2 

90 

-- 

Programming and Programming Languages 

26 

9D 

— 

office Automat l on /AIM. Pus 1 ness 

4 

9E 

-- 

Software Management., systems Software 

2 

97s 

-- 

Other Computer llafdwam and Software 

7 


1 


$54,880.00 
$37,704.00 
$325.00 
$2,550.00 
$ 0.00 
$390.00 
$13,136.44 
$33,332.00 
$960.00 
$ 0.00 
$24 . 871 .00 
$3,050.00 
$500.00 
$8,084.00 
$320.00 
$ 0.00 
$22,797.88 
$634.00 
$1,009.00 
$750.00 
$99.00 
$ 0.00 
$131.00 
$386 

$1,418.W' 
$2,438.00 
$595.00 
$ 1 , 210.00 
$1 . 255.00 
$2,500.00 
$1,127.00 
Cn no 


TOTALS l r ()U AC -- Comptroller 


621 


216 , 532.32 


T 
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2.1.3 Report # 3 -- Summary by Organization 

SELECT DISTINCT Reportl.unit, COUNT(Reportl.coursenum),; 

SUM(Rcport 1 .cost), Unils.unitname; 

FROM Report 1, Courses, Units, Traintyp; 

WHERE Courses.coursenum = Report l.coursenum; 

AND Units. unit = Reportl.unit; 

AND Traintyp.tot = Report 1. tot; 

GROUP BY Reportl.unit; 

ORDER BY Reportl.unit; 

CMATCMTORT FROM Rcport3 WIDTH MO COLUMN NOOVERWRITE 

REPORT FORM report3.frx TO PRINTER NOCONSOLE NOEJECT 


2.1.4 Report # 4 - Summary by Type of Training 


SELECT DISTINCT Reportl.unit, COUNT(Reportl.course num),, 
SUM(Reportl .cost), Unils.unitname; 

FROM Report 1, Courses, Units, Traintyp; 

WHERE Courses.course_num = Report l.course_num; 

AND Units.unit = Reportl.unit; 

AND Traintyp.tot = Rcportl .tot; 


GROUP BY Report 1. tot; 

ORDER BY Report l tot; 

FROM RCPON4 WIDTH 040 COLUMN NOOVERWRITE 
REPORT FORM report4.frx TO PRINTER NOCONSOLE NOEJECT 


2.2 On Training Costs and Priorities 

2.2.1 The Financial Impact of Training 

Training is expensive, however, lack of training could be even more expensive! Reliable 

So" 

"ROOKS" and "OTHER COSTS" store information about each course s cost. Ui - v< There 

bases. 1 

. Direct Materials, instructor, tuition, cost for media, cost of employee travel and per diem, 
employee salary during training, opportunity cost or foregone production. 

. Indirect Administrative, ofTice space, computers, simulators, depreciation of facilities. 

. Intangibles. Costs associated with potential failure of personnel to perforin a task or job, savings 
associated to finishing a job early due to good performance by everybody involved. 


« The terms "direct" and "indirect"costs refer here ,o its ^ 

"direct” cost may be understood as the cosl borne by the trainees organization, while .nd^ect ts.nyc 

organization. 


r*u. rrrvr d n-r Aucnid 3 1993. 
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07 / 15/93 


1994 Individual Development Plans 

Summary by Organization 


Page 


Organization 


Number of Requests 


Cost 


-- No mall code provided 
AC -- Comptroller 
CC -- Chief Counsel 
CD -- Center Director 

CM __ payload Management and Operations 
C p -- payload Projects Management 
__ sxs Payload Operations 
CV — Expendable Vehicles 
DE — Engineering Development 
j-jp — Facilities Engineering 
DL> __ Electronic Engineering 
DM -- Mechanical Engineering 

EO -- Equal Opportunity Program 

£ X __ Executive Management 

HH __ Human Resources and Management Systems 

MD __ Biomedical Operations and Research 

MK -- NASA Headquarters /Space Shuttle Operations 

OP -- Procurement Office 

PA — Public Affairs Office 

PM -- Personnel Office 

PT — Technology and Advanced Projects Office 
EM — Mission Assurance 
ro -- Ouality Assurance 

rq -- Reliability and Quality Assurance 
yiy -- safety and Reliability 
SI -- Center Support Operations 
ss __ space Station Project Office 
TE -- Grounds Engineering 

TL -- Shuttle Logistics Project Management 

TM -- Shuttle Management and Operations 

TP -- Shuttle Operations 

TV -- Vehicle Engineering 


Totals for KSC 


13 

$4,764 .00 

621 

$216,532.32 

7 

$2,389.00 

20 

$23,616.00 

40 

$18, 154.00 

318 

$193,895.11 

2318 

$1,321,368.35 

170 

$121,802.00 

136 

$59,804.00 

900 

$484 , 378.92 

1033 

$572,131.68 

1225 

$459,930.46 

28 

$5,742.00 

55 

$19,647.00 

12 

$5,607.00 

294 

$148,244.18 

156 

$55,289.00 

710 

$301,214.91 

221 

$126,324.01 

733 

$103,086.00 

154 

$81,938.00 

490 

$294,545.' 

2464 

$548. 692. lV"' 

50 

$30,072.66 

659 

$226,156.10 

1254 

$544,933.96 

262 

$159,045.00 

985 

$811,079 .95 

363 

$180,313.00 

471 

$267,918.92 

842 

$338,121.95 

3066 

$1,388,009.52 

20070 

$9,114,747.17 
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07/27/93 


Page 1 


1994 Individual Development Plans 

Summary by Type of Training 

Number of Requests 


Type of Training 


10 

Executive and Management 

20 

Supervisory 


3A 

Legal 


3B 

Medical and Biomedical 

3C 

Physical Sciences 

3D 

Earth Sciences and Natural Resources 

3F 

Human Factors 


3H 

Mathematics and Statistics 

3 J 

Engineering - 

General 

3K 

Engineering - 

Aeronautics 

3M 

Engineering - 

Aerospace 

3N 

Engineering - 

Chemical, Petroleum, or 

3 P 

Engineering - 

Civel , Architectural , 

3Q 

Engineering - 

Electrical /Electronic 

3R 

Engineering - 

Mechanics /Mechanical 

3S 

Engineering - 

R & QA 

3T 

Engineering - 

Systems, Industrial 

3U 

Engineering - 

Safety 

3W 

Other Legal, 

Medical and Scientific 


Cost 


4A 

4B 

4C 

4D 

4E 

4G 

4J 

4K 

4N 

4P 

4Q 

4R 

4S 

4U 

4Y 

4Z 

5A 

5B 

5C 

6A 

6B 

6C 

6D 

7B 

7 Z 

8A 

8B 

8C 

8E 


Finance, Accounting, Auditing 
Personnel 

EEO, Race or Ethnic Studies 
Procurement 

Policy, Program, or Management Analysis 

Quantitative Analysis or Operations 

Safety Systems and Operation 

Security or Investigation 

Logistics 

Supply 

General Safety and Health 
Foreign Language 

Program and Project Management 
Sales, Marketing and Customer Service 
Other Administrative 

Systems, Safety and Area Access Training 
Academic Course Work 
On-the-Job Training 
Basic Clerical 

Inter-personal and Office Management 
Administrative Systems 
Other Clerical 

Engineering and Science Support /Journeyman 
Other Trade, Craft, Apprentice and 
Orientation 
Career Planning 

Pre-Retirement FilcFiN_ 

Nets/Quality Circles 


1211 

$1,892,420 

2691 

$1,655,615 

31 

$17,805 

33 

$4,620 

19 

$6,655 

112 

$65,637 

15 

$0 

5 

$178 

186 

$44,420 

33 

$4,322 

698 

$391,765 

146 

$ 56,735 

190 

$115,939 

473 

$358,319 

165 

$97,324 

246 

$128,603 

55 

$ 72,268 

100 

$58,419 

46 

$5,565 

1224 

$138,947 

249 

$75,729 

291 

$423,196 

272 

$16,439 

745 

$188,475 

53 

$14,938 

32 

$5,226 

34 

$12,407 

84 

$12, 470 

59 

$22,669 

20 

$3,345 

288 

$59,557 

14 

$6,876 

1065 

$836,679 

22 

$5,545 

86 

$27,610 

1325 

$0 

2050 

$552,941 

24 

$0 

109 

$9,086 

354 

$35,828 

242 

$30,297 

107 

$9,989 

39 

$4,945 

7 

$7 , 000 

24 

$0 

243 

$6,836 

, August 7, 5993, page 8 

$1,188 

472 

$29,117 


441 


Flic: Reporlf, Summary hy Type of Training 



07/27/93 


Page 2 


1994 Individual Development Plans 


Summary by Type of Training 

Type of Training Number of Requests 


Cost 


8F 
8 G 
8H 
8Z 
9A 
9B 
9C 
9D 
9E 
9F 
9Z 


Productivity 

Professional /Personal Development 
Communication Skills 
Other General 

Telecommunications , Networking 
Programming and Programming Languages 
Engineering Applications 
Office Automation/AIM, Business 
Software Management, Systems Software 
Artificial intelligence, Expert Systems 
Other Computer Hardware and Software 


67 

$73,907 

943 

$156,450 

1117 

$271,057 

85 

$68,633 

117 

$95,060 

764 

$710,244 

146 

$106,807 

586 

$66,532 

71 

$40,547 

6 

$1,875 

104 

$9,690 


TOTALS FOR KSC 


20070 $9,114,747 
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2.2.2 The Priority of Training 


PTDS' IDP.DBF database file currently stores a "PRIORITY" field for =-Tb 'raining rcquest. 
Unfortunately, the field refiects the order in which the employee listed tato ’nSrflect the 

course listed is assigned priority 1 , the second one has priority 2, etc ). Obviously this does not 
potential impact oftlie training for the Center, nor for the person making the request. 


2.3 Recommendations for Further Work 
2.3.1 On the Financial Impact of Training 

Ensure *h a * otwm'asstSalod'fo'eadtWini'ng rourse. 

^ te r^nt g r„;n“r, 1 r ; 

Branch. Tlie cost data base should store for each course, he t a g ( ^ computer will 

resources travel requirements, etc ), rather than the actual dollars spent. At report time v 

2.3.2 On the Priority of Training 

Include in the data base a field that refiects the 5JS ""“" 8 

has for the Center and for the person making * j, may be the best method to 

trainee. 

As a possible indicator of priority of content and timing. *££*(£*• — — “ 
Summer of 1992 in the Needs Assessment effort conducted by the author at KSC. 


Potential Impact 


Weight 


Helps organizational performance 
Helps individual’s performance in current position 
Enables person to perform more responsible position 
Useful but not essential 

There is no need for training in this subject matter 


4 

3 

2 

1 

0 


Priority on Timing 


Weight 


Needed within next three months 
Needed within next six months 
Needed within next year 
Not an urgent need 


4 

3 

2 

1 
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3.0 THE EVALUATION OF TRAINING 


3.1 Existing Evaluation Procedures 

Evaluation or training courses is limited to KSC Form 13 (see next page). This form is processed 
manually and II* major problems am H,a. 11 is not linked to KSCs goals and object, ves. lh e,e a,e no 
cost/bcnefits considered, the form is not used to assess performance .mprovement, and the results are not 

used for formal reporting. 


3.2 Objectives 

The evaluation of training at KSC is envisioned as having the following properties: 

• Linked to KSC Mission, Goals and Objectives. 

• Has a performance and vatucs-based focus. 

• Measures results in terms of performance improvement, financial impact productivity 
and quality improvement, team building accomplishments, and less emp oyee urnover. 

• Resets priorities based on results. 

• Tracks training and performance data. 

TJie link to KSCs mission, goals and objectives and the performance and values-based focus will 
be ascertab^'Ihrough user invelvenmnl I, is lire user (NOT Ihe Human Resources Dev. opmen. Branchl 
who must define wlral is "required level of pcrlbrmanee" The evaluation process '^**?*p ^ 
training's impael on performance by asking the users: "Whal is the expected result of tramrng? and 

"What may happen if no training is given?” 


3.3 Evaluation: A Literature Search 

A literature search was performed to ensure that the proposed evaluation wi«i 

standard practice. The following sections discuss the sources, most common 

methods to conduct the evaluation and an indicator of performance effectiveness, the Content Validity 
Ratio" A detailed discussion of the search findings is provided in the appendix, along with the list 
rcferenccs used Copies of all references were submitted to NASA in a separate binder and are considered 

part of this final report. 


3.3.1 


Sources 


The literature search yielded 782 hits when the keep words "TRAINING", "EFFECTIVENESS , 
"ASSESSMENT" and "EVALUATION” were used. From those 782 titles, a total of 78 abstracts were 

of those abstracts resulted in 22 promising articles of which .14 were 
rJanaWe at UCFs library (or though inter-library loans). Most articles are rather theoretical in nature but 

a few described specific applications in government agencies such as the ^P art, " en ‘ ° f ^menfo^the 
Protection and Quarantine and Forest Service divisions), NASA Headquarters, the Department of 
Navy’s Navy Finance Center, and the Office of Personnel Management. 
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PARTICIPANT'S EVALUATION 



PARTICIPANT’S NAME 



SUPERVISOR'S NAME 


PLEASE INDICATE YOUR EVALUATION OF THE SEMINAR IN TERMS OF THE FOLLOWING: 


EXCELLENT 


|, Ovcrolt «*oluo lion of nwinoi, 

1 . S.rmnor content - omoont of detoil ond extent of covroge. 


3. Level of pre *entotlon. _ _____ 


4. Effectivenee* ol hond-oul molt __ 


5. E llecti venei* of oudio/vimol oicft. __ 


6. lnil»uclor( i) knowledge of tubjecl. ______ 


7. Inetruc loi( t) eflectiveneM. _______ 


8, Applicotion lo your job. 



COMMENTS: 
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3.3.2 Approaches 

Training evaluation usually takes 

conducted at the end of the training exercise . . tKre lhc evaluation focuses on whether 

environment, instructor, and the Mml .fl^^h S^S^t the subject taught. (3) 

Bdwvhml'whme t^ e^uarion consists on following the emphasis ofttw'** 3 ' 

e ** s ^ r **-* ^ « ,te oreama "' 0 "' 5 

benefits accnied as a result of the training. 

3.3.3 Methods 

to achieve as a result of the training and . i nau i r ies and the PAPA method are that they 

attained. The main problems with observations ^! . Q d ’ oney) lo conduct. For an on-going 

may lack objectivity and require vast resources the best 

^f sSrdiXU which printing, distribution, collection, analysis and 
reporting can be mechanized. 

3 3 4 Content Validity Ratio (CVR) 
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3.4 The Evaluation Procedure 

The proposed evaluation procedure consists of a set of two evaluations and three reports. Copies 
of the evaluations (as proposed and as implemented) are included in the next pages. 


3.4.1 The Scanning Process 

AnOpScan Model 5 fro,,. National Computer Systems (NCS) »mt 

mechanize ft, .pressing of ,« to 
DESIGN, a Windows based graph** in ro,ms supplied by 

3.4.2 Evaluations 

The proposed Endof-Course Evaluation is a "reaction" type evaluation. It has four sections. The 
firs, one, 'Externals,- deals '/SUenC 

^ta^rpcS value for’ the organization. An overall 'Happiness Index' 
^^teaKtKia^^as^tm aCiVR type of analysis) to the trainees' reaction to each training course. 

A Follow Ud Evaluation was proposed, to be delivered about six months alter the training. Both 
supervisors' delayed assessment of the real impact or each training course. 


3.4.3 Reports 

under his supervision. 

An End-of-Quarter Report is planned which will summarize the number of courses taught, the 
number of students, cost, and indicators of overall happiness and overage * jenc^ ^ ^ ^ ^ , , 
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KENNEDY SPACE CENTER 
TRAINING BRANCH 

INITIAL EVALUATION OP TRAINING 


SSN : 

- 

HAIL CODE: 



C0UR9E ; 
INSTRUCTOR i 
COURSB DATB(S) t 
SITE « 


INSTRUCTIONS : Indicate the degree to which you agree with the following statements using 
gA=STRONGLY AGREE A= AGREE D.DISAGREE SD=STRONGLV DISAGREE »A=NOT APPLIC 


EXTERNALS 

1 The environment was suitable for learning. 

2. There were no distractions during the session. 

3 Handouts used at the session were effective. 

4. Audio/visual aids used at the session were effective. 

INSTRUCTION 

5. The instructor delivered the lessons clearly and effectively. 

6. The instructor was knowledgeable of the materia taug 

7 The instructor kept my interest throughout the session. 

8. ' The instructor showed concern lor the student's understanding 

of the material. , 

9. The hands-on component was delivered effec iv y. 

10. Enough time was alloted to hands-on practice. 


SD NA 




FOCUS 

11. The material in this class is consistent with the actual and 

current requirements of my job. 

12. The training focused on the specific tasks, knowledge an 

skills needed for acceptable job performance. 

13. The knowledge and skills were taught to the appropriate le 

of proficiency. 

14 Training was included that is not needed on the Dob. 

15. Required skills and knowledge were not adequately covered. 

16. The level of knowledge attained in training meets the level 
needed for acceptable job performance. 


ooooo 

ooooo 



ooooo 


ASSESSMENT 

17. Overall, I am pleased with this course. 

18. This training will help me on my job. 

19. This training course is right on target with KSC s 

20. I reconmend that others from my organization attend thio 


needs . 
course. 
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KENNEDY SPACE CENTER 
HUMAN RESOURCES DEVELOPMENT BRANCH 

INITIAL EVALUATION OF TRAINING 


o 


NAME: 


COURSE i 
INSTRUCTOR i 
COURSE DATE { S ) * 
SITE i 


INSTRUCTIONS: Indicate the degree to which you agree with the following statements using 
SA=STRONGLY AGREE A= AGREE D=DISAGREE SD=STRONGLY DISAGREE NA=NOT APPLICABLE 


PLEASE COMPLETELY FILL IN THE APPROPRIATE BUBBLES WITH A #2 PENCIL. 


EXTERNALS 

1. The environment was suitable for learning. 

2. Handouts used at the session were effective. 

3 . Audio/visual aids used at the session were effective. 



INSTRUCTION 

4. The instructor delivered the lessons clearly and effectively. 

5. The instructor was knowledgeable of the material taught. 

6. The hands-on component was delivered effectively. 



FOCUS 

7. The material in this class is consistent with the actual and 
current requirements of my job. 

8. The level of knowledge attained in training will help improve 
my job performance. 


ooooo 

ooooo 


ASSESSMENT 

9. Overall, I am pleased with this course. 

10. This training course is consistent with KSC's needs. 

11. I would recommend that others from my organization attend 
this course. 

Suggestions for improvement or other comments: 



OOOOO 
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KENNEDY SPACE CENTER 
training BRANCH 

FOLLOW-OP EVALUATION OF TRAINING 



SSNf 

- 

MAIL CODEt 



COURSE t 
INSTRUCTOR! 
COURSE DATE (S) t 
SITEt 


INSTRUCTIONS : Indicate the degree to which you agree with the following statements using 
^STRONGLY AGREE A=AGREE DISAGREE SD=STRONGLY DISAGREE »»=NOT APPLICABLE 



on the basis o£ this employee's performance after the referred 
training course, the following have improved: 


1. employee’s supervisory and management skills 

2. personal productivity. 

3. organization’s productivity, 
i. employee's satisfaction. 

wwj . customer's satisfaction. 


6. job performance-quality. 

7. job per formance -quant i ty . 

8. organization's morale. 

9. safety. 

10. stress reduction. 

11 . team building . 

12. cost awareness and control. 

13. environmental awareness. needs 

14 This training course is right on target with KSC s needs. 

1 5 '. 1 racomand that oth.r. from my orj.ala.tlon attend thi. couraa. 


SA A D SD NA 



THIS FORM WAS FILLED OUT BY: 


EMPLOYEE 

SUPERVISOR 


TODAY’S DATEt 
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r nt rn Number 75612-5-R5 


Page 1 

SURVEY NETWORK® 

4S1 


10098 




TO: 


(SUPERVISOR'S NAME} 


MAILCODE: 


o - 


KENNEDY SPACE CENTER 
HUMAN RESOURCES DEVELOPMENT BRANCH 
FOLLOW-UP EVALUATION OF TRAINING 


PARTICIPANT'S NAME: 


COURSE t 
VENDOR: 

COURSE DATE ( S ) 


degree to Shich you agree with the following statements using: 


SA= STRONGLY AGREE 


A=AGREE D=DISAGREE SD=STRONGLY DISAGREE NA-NOT APPLICABLE 



1. This employee's job performance has improved as a direct 
result of this training. 

2. I would recommend that others from my organization attend 
this course. 

TODAY'S DATE: 

Suggestions for improvements or other comments concerning 
this training: 


ooooo - 
ooooo - 


PLEASE RETURN THIS SURVEY TO HM-PER-1 IN A U.S. GOVERNMENT 
MESSENGER ENVELOPE. 
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3.5 Expectations 

The proposed evaluation procedure will (a) exan,i " C ^ and 

evaluation of the results obtained (aggregate values). 


3.6 


Recommendations for Further Work 
3.6.1 On Evaluation: 


( J UII LVitiunuv... 

Tlic first recommendation is to implcnK^ll“ (o loduc e standard 

report recipients. 


3.6.2 


On tire Training Management Information System: 


^ 11 U«V * - D - 

. PTDS (NTDS?) information 
The second recommendation is to link ^e ^« al, ° n hours, units of resources etc.) (b) 

data. 


3.6.3 On the Needs Assessment: 


l VMi l"'" 

The whole Training System should be (Needs Analysis), how 
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4.0 APPENDIX: Literature Review 


The following p»gc S »c,e propped ^,„rt. e 

SBS2SEE ~~±£ss=r-~ 
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TRAINING EVALUATION - An Overview Of Existing Practices 

Four categories of training evaluation are reaction, 
learning, behavior and results [Stevens & Hellweg, 1990]. 

Reaction is the trainee's liking of the training 
course. This response should be determined as soon as 
possible after the session is completed [Dewine, 1987]. 
Questions dealing with the atmosphere, physical environment, 
instructors and training media should be included on the 
first questionnaire. An example of a reaction based 
question is: 

6. Rate the use o£ handouts in the training. _Excellent .Good .Fair .Poor 

Learning is the evaluation of whether the trainee 
remembers (not necessarily uses) facts, principles and 
behaviors taught in class. This is the least relevant 
level of evaluation for an organization because increased 
knowledge does not necessarily mean the knowledge is used on 
the job [Stevens & Hellweg, 1990]. An example of a learning 

based question is: 

2. Pentium class microprocessors must be super-cooled prior to use. .True .False 

The behavior level is the most complex level of 
evaluation. This deals with the actual job improvement of 
the trainee. The behavior evaluation should come from 
trainee, its supervisor, subordinates, and peers. It should 
be performed at least three months after the training 
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[Stevens & Hellweg, 1990]. An example of a behavior based 
question is: 

6. I am capable of speaking to large groups: .Excellently _Well ^Acceptably .Poorly 

Results are the actual effects that training has had on 
an organization. Items such as turnover, absences, sales, 
customer satisfaction, quantity and quality of performance, 
morale, cost reductions, grievance reductions, accident 
rates, suggestions, employee satisfaction and time use 
should be evaluated [Stevens & Hellweg, 1990]. 

8. Morale at the worksite has (since my training). _lncreased significantly 

.Remained about the same 
.Decreased significantly 

The outcome of one of the four areas may affect other 
areas [Dewine, 1987] . The amount of time between training 
and a post training evaluation is not a standard. Ranges of 
suggested times are from two weeks to twenty-three months 
[Stevens & Hellweg, 1990] . An average time is from four to 
six months. 

An alternate method of evaluation of training is 
content evaluation. Content validity is the evaluation of 
how the training content is related to the job. The 
training content must be identified. Items that are 
included are knowledge, skills, abilities, and other 
personal characteristics (KSAOs) that are needed for job 
performance . The content is then evaluated through a Content 
Validity Ratio (CVR) approach. People who have a thorough 
knowledge of the job rate each KSAO on its importance to the 
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job. The following formula is used to determine validity 
[Ford & Wroten, 1984]. 

cvr - ppnnU stating KSAO is important. - Peop l e stat i ng KSAQ i s unimporta n t . 

Total Number of: People 

Positive CVRs mean that greater than half of the people 
stated that the KSAO was important and negative CVRs mean 
that less than half of the people stated that the KSAO was 
important. A method for matching the CVR data and training 
needs would include the following steps. First divide the 
KSAOs into categories. Have your job experts rate the 
important KSAOs on a scale. This scale would determine how 
training intensity and resources should be' spent on 
different areas of subject content [Ford & Wroten, 1984] . 
These methods would work well. However, they would require 
customized forms for each course and extra manpower because 
the procedure could not be automated. 

Kruger & Smith (1987) recommended two different 
surveys: a skills/behavior survey and a health/ stress 
survey. Both surveys use a system of examining statements 
and choosing to what degree you agree with the statements. 
The following is an example: 

3 . i am able to communicate with my manager. AGREE 987654321 DISAGREE 

The skills/behavior survey deals with what was 
learned, how the trainee's performance has changed and how 
it has affected the organization. A survey is filled out by 
the trainee, its supervisor and some others. A sample 
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question on a skills/behavior survey might be: 

6. I am capable of speaking to large groups: _Excellently _Well _Acceptably _Poorly 

The health/stress survey identifies health and stress 
risks to the employee. It is only be filled out by the 
trainee. A sample question on a health/stress survey might 

be: 

6 My knowledge of safety that is required for the job is: _Proficient 
3 _ Adequate 

_Not Adequate 
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Existing 


Training Evaluation Programs 

t- raining evaluation programs 
The following are some training ev 

i =.t- different governmental and non- 

currently in place at different y 

governmental agencies. 

Department of Agriculture-Plant Protection and 
ouarantine: The Department of Agriculture's Plant Protects 

and ouarantine branch currentlv uses an evaluation program 

in which the evaluation is done by front-line supervrsors. 

, -in t-vw=> Skills needed to 
nre first trained in tne sku 
T he supervisors are 

evaluate the trainees. The supervisors are then assigned to 
evaluate recently trarned officers. The evaluators may use 
questionnaires , interviews, performance observations an 
miscellaneous inguiries to gather data. These are all use 
aEter the training. The trainees supply data concerning 
their performance through either guestionnaires and 
interviews. The evaluators gather their own data through 
on-the-job performance evaluations and candid inquiries^ T 
data is collected by a central organisation and compile 

if the skills taught actually assisted the 
determine if the sxm 

th their jobs. Advantages of this program mclu 
officers with their 

i ant data for evaluation and actively 
providing relevant data 

involving supervisors in the evaluation. Disadvantages 
include long length of time needed for process, requiring 
supervisors to do wort soiely on evaiuation causes need t 
hire temporary replacements and expensrve travel costs o 
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supervisors. A final disadvantage is that the evaluation is 
based on the specific job or supervisor's objectives and not 
those of the entire organization [Salinger & Roberts, 1984] 
Department of Agriculture- Forest Service: The Forest 
Service currently uses a questionnaire-based evaluation 
system. Standard questionnaires are completed by the 
trainee, its supervisor and subordinates before the training 
session and four weeks after the session. Copies of the 
questionnaire are also completed by untrained personnel and 
their supervisors and subordinates in order to have a 
control population. Questions in the questionnaire are 
behavior based such as the following 


To what extent do you think training: 

Will improve your ability to be a better supervisor 
Will increase productivity? 

Is a good use of your time? 

Improve your communication skills on the job? 


GREATLY NOT AT ALL 

987654321 
987654321 
987654321 
987654321 


Results are statistically tabulated. The results are then 


compiled and reported to the specific organizations. The 
questionnaires are behavior based and completely 
confidential. The tabulation stage tests the statistical 


significance of questions, trainee, supervisor and 
subordinate perceptions and comparison of trained results 
versus the control's results. Advantages of this program 


include accurate results due to use of a control group and 
small amounts of time and resources needed. The only 
significant disadvantage is the possibility of low return 
rate of questionnaires [Salinger & Roberts, 1984] . 
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Administration: The 


National Aeronautics & Space 

evaluation program described below was for a career seminar 
sponsored by NASA Headquarters. The evaluation was directly 
built into the training program. The evaluation consisted 
both pre and post training questionnaires, group interviews 
and individual interviews. The questionnaires contained 
general, open-ended questions dealing with current and 
future dob goals. A sample question on a survey might be: 

, „„„ h... you done to .ch«»= you, In th. P~t 30 d.y»7 

In the group interviews, the trainees spote about their own 

job plans and how they were using what they learned. 

Trainees could hold optional individual interviews with the 

instructor concerning their own progress. There were also 

informal evaluations during the course. The instructor was 

the only evaluator involved and there was no major 

computation of data in the evaluation. The instructor had to 

be available after the training session for many months in 

order to do the personal interviews. Advantages of the 

evaluation program included adaptability. Disadvantages 

include the fact that it is not very structured, forms need 

to be customized for different courses, constant need for 

the instructor, no organizational input and most of worb has 

to be in groups which may be difficult to assemble [Salinger 
& Roberts, 1984] . 

Department of the Navy- Navy Finance Center: The Navy 
Finance Center evaluated one of its Interaction Management 
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Courses through a system of questionnaires done in a group 
setting. The questionnaires were completely behavior based 
with questions such as: 

When my supervisor speaks to me regarding disciplinary action , he/she . . . 

Always Never 

Uses my ideas when I offer them. 654321 

Acknowledges my feelings about the situation. 654321 

The questionnaires were completed by employees, their 
managers and subordinates that did and did not take the 
course. The evaluation was done before the training and two 
years after. The results were then computer tabulated to 
create statistical averages of how the training works 
through the perceptions of the employees, their managers and 
subordinates. Reports were then created from that data. 
Advantages of this evaluation method includes involvement of 
all members of an organization, use of a control (the 
untrained sample) to ensure accuracy of results and high 
return rate because of the group approach to completing the 
questionnaire. Disadvantages include a management commitment 
and travel expenses if the respondents are not in a common 
geographical area [Salinger & Roberts, 1984] . 

Office of Personnel Management: The OPM devised a 
generic training evaluation program called the Participant 
Action Plan Approach (PAPA) . The evaluation system asked 
each participant to identify certain behaviors or actions 
related to the course that he/she would like to change. Here 
the evaluator would either interview the trainee (by phone 
or in person) or use questionnaires (six months after the 
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training) to determine whether the actions were carried 
out. The evaluation also checked for results of tried action 
plans and why untried action plans were not attempted. The 
questionnaire also asked how the course could be changed to 
better teach the trainee. Problems and obstacles that 
deterred the trainee from carrying out the plan were also 
identified. The result were computer tabulated. Results were 
used to create reports and eventually modify the course. The 
program provides data primarily concerning behavioral 
changes brought about by the course. Advantages of this 
program include ability to be used for many courses and it 
can accurately describe what actions came from the training. 
Disadvantages include the need of self-reports and skilled 
interviewers, large amount of time needed and it does not 
involve supervisors and subordinates in the evaluation 
process [Salinger & Roberts, 1984 ]. 


Flic: WRTUrTOT.DOC, 12 July, 1993, page 9 


463/464 



1993 NASA/ASEE SUMMER FACULTY FELLOWSHIP PROGRAM 


JOHN F. KENNEDY SPACE CENTER 
UNIVERSITY OF CENTRAL FLORIDA 


*** report NOT AVAILABLE AT PUBLICATION TIME *** 


PREPARED BY: 

ACADEMIC RANK: 

UNIVERSITY AND DEPARTMENT: 

NASA/KSC 

DIVISION: 

BRANCH: 

NASA COLLEAGUE: 

DATE: 

CONTRACT NUMBER: 


Dr. Shcrif A. Sherif 

Associate Professor 

University of Florida 

Department of Mechanical Engineering 


University of Central Florida 
NAS A-NGT -60002 Supplement: 11 


PMCHDtNfi PAGE BLANK NOT FILMED 


465/ 466 




